Target recognition and pattern classification involves the evaluation of new observations on the basis of past observations to distinguish targets or desired patterns from background clutter. This task is complicated by the complex nature of real world environments. Moreover, the computational overhead of many practical classification problems strain serial computer resources.
With respect to the complexity of pattern recognition and pattern classification techniques, many of these techniques rely on feature information to differentiate preferred targets from background clutter. Coordination and selection of this feature information is an ongoing problem, as the desired feature information itself (e.g., contrast-based and texture-based) often varies from target to target (e.g., size, stationary, make-up, composition, orientation), target to background (e.g. location of target), data set to data set (e.g., lighting, time-of-day, ambient temperature, and context of scene, sensor characteristics), and data source to data source (e.g., one-dimensional or multi-dimensional data, such as digitized infrared imagery, digitized TV imagery, digitized Infra-red imagery, speech samples, or radar samples).
Recently, researchers have focused on the perception of features in the visual field by biological visual systems and the organization of these biological visual systems. In short, this research has focused on what features are perceived and how are the receptive fields (e.g. neurons sensitive to particular features) organized. This work contributes to the understanding of biological vision systems and lays the groundwork for artificial vision systems in such areas as robotics and automatic target recognition.
In particular, D. J. Hubel and T. N. Wiesel in their article Hubel, D. J. and Wiesel, T. N, “Receptive Fields, Binocular Interaction, and Functional Architecture in The Cat's Visual Cortex.” Journal of Physiology. vol. 160, pp. 106-154 (1962) and D. Marr in his book Marr, D. Vision W. H. Freeman and co. San Francisco 1982 are credited with initiating research in this field. Recent work, however, in artificial neural networks suggests mechanisms and optimization strategies that explain the formation of receptive fields and their organization in mammalian vision systems. This work includes Miller, K. d., Keller, J. B. and Stryker, M. P. “Ocular Dominance Column Development: Analysis and Simulation.” Science, vol. 245, pp. 605-615 (1989), Durbin, R. and Michison, G. “A Dimension Reduction Framework For Understanding Cortical Maps.” Nature, vol. 343, pp. 644-647 (1990), Linsker, R. “Self Self-organization In a Perceptual Network.” Computer vol. 21, pp. 105-117 (1988), and Kammen, D. M. and Yuille, A. L. “Spontaneous Symmetry-breaking Energy Function.” Biological Cybernetics, vol. 59, pp. 23-31 (1988). Linsker demonstrated how Hebbian learning algorithms, which change synaptic connections according to the degree of correlation between neuronal inputs and outputs, give rise to layers of center-surround and orientation-selective cell, even if the input to the initial layer is random white Gaussian noise. Kammen and Yuille show that orientation-selective receptive fields can also develop from a symmetry-breaking mechanism. Under certain conditions, the receptive fields perform a principal component analysis of the input data, as was shown in Oja, E. J. “A Simplified Neuron Model As A Principal Component Analyzer.” Mathematics and Biology, vol. 15, pp. 267-273 (1982). Similarly, the article by Brown, T. H., Kairiss, E. W., and Keenan, C. L., “Hebbian synapses: Biophysical mechanisms and algorithms,” Annual Review of Neuroscience, vol. 13, pp. 475-511 (1990) suggested Hebbian learning occurs in nature.
With respect to the computational requirements, neural networks provide parallel computational implementations. These networks embody an approach to pattern recognition and classification based on learning. Example patterns are used to train these networks to isolate distinctions between the particular patterns and background clutter for proper classification.
With respect to neural networks, the architecture of a neural network can be simply represented by a data-dependency graph, such as that shown in FIG. 1. As shown, this data-dependency graph includes a set of active nodes 4 and a set of passive communicative links 6. The graph nodes 4 represent artificial neurons and the passive communication links 6 define unidirectional communication paths between the artificial neurons 4 (graph nodes 4). Additional links 2 are used as network inputs, which indicate which neurons will receive input signals from outside the network. Similarly, additional links 8 are used as network outputs, which indicate which neurons will transmit output signals to outside the network. A communication link 6 effectively makes the output state of one neuron at the tail of the link available to the neuron at the head of the link. For example, if the neuron states were made available as electrical voltages or currents, the communication links could be provided as simple conductive wires.
Referring to FIG. 2, some architectures are feed-forward, wherein signals are inputed through communication links 2 which direct the input signals, flow through the communication links 6 to the neurons 4 in the neural network in a layer-by-layer fashion, and outputted through the communications links 8 which direct the output signals. No circular path of feedback exists within this type of network. Other networks are fully interconnected, wherein all neurons are connected to all other neurons. FIG. 3 shows an example of this type of network, wherein all neurons 4 are connected to all other neurons 4 through corresponding communication links 6 with input and output communication links as shown. In general, each neuron in a neural network architecture has a high fan-in and a high fan-out.
With respect to the neuron function itself, neuron states are generally defined by non-linear local functions. The output value of a neuron is defined by a transfer function, which depends only upon the neurons' current internal state, a set of local parameters called synapses, and the value of signals received from other neurons. FIG. 4 shows a simplified schematic of a neuron, along with its network links 12, synaptic parameters 14, and typical input-output transfer function 16. Input voltages V1, V2, V3, . . . VN are received from other neurons and output voltage Vi is outputed to other neurons. The neuron transfer function 16 is usually described as a non-linear function of the sum-of-products of the synapse parameters 14 with their associated input signal values. FIG. 5 shows a typical non-linear function that must be provided by each neuron that operates according to equation 1, which is defined as follows       V    i    =      F    ⁡          (                        ∑                      j            =            1                    N                ⁢                                   ⁢                              σ            ij                    ⁢                      V            i                              )      In defining the transfer function described by equation 1, Vi is the output value (or state) of neuron i,   ∑  ijis the value of the synapse parameter that modifies the effect of the output of neuron j on the state of neuron i, and F is a function of the sum-of-products of the synapses with neuron states. With respect to FIG. 5, F is a linear function of the sum-of-products input near the zero sum region, while saturating to positive and negative values when the sum exceeds a preset positive or negative value.
With respect to training and using the neural network, the synapses affect the transfer function by modulating the strength of signals received from other neurons in the network. In virtually all models, the behavior of the network as a whole is changed by altering the values of the synaptic parameters. Once the parameters are altered, inputs can be applied to the network, and the processed output taken from the designated set of neurons. The time during which the synapse parameter values are altered is generally called the Training Phase. The time during which input signals flow through the network while the synapse parameters remain fixed is called the Recall or Relaxation Phase.
With respect to neuron circuit embodiments, FIG. 6 shows the schematic diagram of an artificial neuron implemented using resistors and an operational transconductance amplifier. The resistors Rij are associated primarily with the synapses in equation 1, while the operational transconductance amplifier (OTA) and its feedback component R6 is used to provide the desired transfer function Fi.
With respect to the operation of an OTA, as shown in FIG. 6, an ideal OTA generates an output voltage proportional to the difference of the currents entering its inputs, multiplied by a very large gain factor. FIG. 7 shows an ideal OTA. The defining equation 7 for this circuit may be derived by the analysis, as followsVOUT=A(i2−i1) A>>>1 i1=i−+i0 
Vout=A(i+−i−−i0)       i    o    =            V      out              Z      F      
Thus,       V    out    =                              Z          F                ⁢        A                    A        +                  Z          F                      ⁢          (                        i          +                -                  i          -                    )      If A>>ZF,Vout=ZF(i+−i−) The gain factor A is infinite in the idea case, and very large values of A (Over 1 million) can be obtained in practice using off-the-shelf OTA components. An ideal OTA will have a zero input resistance at its inputs, so that the operation of the circuit may be understood by assuming equation 7, and by assuming that the voltage at either input referenced to ground is always zero. Electrical components placed between the output of an OTA and its negative input induce negative current feedback in the circuit while components placed between the OTA output and its positive input induce positive current feedback. Simple resistive feedback causes the OTA to generate an output voltage proportional to the net current difference at its' positive and negative input terminals, as shown in FIG. 8 and the corresponding equations 8, 9 and 10, which are as follows       V    i    =                              R          F1                ⁢                              V            2                                R            2                              -                                    V            1                                R            1                          ⁢                                   ⁢                  V          1                      =                            ∑                      j            =            1                    2                ⁢                                   ⁢                              σ            ij                    ⁢                      V            j                    ⁢                                           ⁢                      σ            ij                              =                        R          FI                          R          j                    
Using these assumptions, the operation of the neuron circuit can be explained. The resistors attached to the input of the amplifier shown in FIG. 6 serve to produce input currents proportional to the product of the input voltages Vj and the conductances of the resistors Rij. The input resistors thus act as the synaptic parameters in FIG. 4, while the neuron state values are represented as the output voltages of the OTAs that implement each neuron function. In general, synapse parameters can be bipolar. That is, these parameters can take on positive and negative values. However, it is not simple in general to implement negative conductances that physically represent mathematically negative synaptic parameters. Fortunately, the differencing nature of the OTA can be used to provide the effect of negative conductances using positive conductance elements. This is accomplished by attaching resistors that provide the absolute (positive) value of the desired conductance to the negative input of the OTA. Since the subtraction of a positive current in the OTA is equivalent to the addition of a negative current, negative synapse parameters can be equally well implemented using standard resistive elements. The feedback resistor R6 defines the slope of the OTA transfer function in the linear region, while the power supply voltages determine the limiting values of the transfer function in regions of strong negative and positive bias. The circuit shown in FIG. 6 provides the basic functional requirements for a typical neural network neuron. Using a more complex feedback impedance in place of R6 can provide more complex transfer functions when necessary.
FIG. 9 shows a schematic of a general neural network based on the use of resistor elements for the synapse parameters and OTAs for the neurons. A non-intersecting grid of wires is used to allow synapse resistors to be connected from OTA outputs to OTA inputs. Pairs of grid lines are used to allow each synapse resistor to be connected from an OTA output to either a positive or negative OTA input terminal. This functionality may be implemented in a software domain as well. For instance, the software algorithm shown in Table 1 produces a dynamic change in neuron state similar to the dynamic changes expected from the OTA array shown in FIG. 9. The synaptic parameters (Resistor values) are assumed to be fixed and pre-stored in a two dimensional matrix R. The OTA feedback resistors are assumed to be the same for all neurons. The neuron states are assumed to be initially preset to certain values and are initially defined by a one dimensional matrix V. All parameters are assumed to be double precision floating point. While only the relaxation model is shown, other models exist and are well known in the art. The training algorithm varies according to the specific neural network type.